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Abstract 

Let the manifold X parametrise a family of compact complex sub- 
manifolds of the complex (or CR) manifold Z . Under mild condi- 
tions the Penrose transform typically provides isomorphisms between 
a cohomology group of a holomorphic vector bundle V — > Z and the 
kernel of a differential operator between sections of vector bundles 
over X. When the spaces in question are homogeneous for a group 
G the Penrose transform provides an intertwining operator between 
representations. 

In this paper we develop a Penrose transform for compactly sup- 
ported cohomology on Z. We provide a number of examples where a 
compactly supported cohomology group is shown to be isomorphic to 
the cokernel of a differential operator between compactly supported 
sections of vector bundles over X. We consider also how the "Serre 
duality" pairing carries through the transform. 

1 Introduction 



Let the real manifold X parametrise a family of compact holomorphic sub- 
manifolds of a complex (or perhaps CR) manifold Z. If certain conditions are 
satisfied, the Penrose transform enables one to interpret Dolbeault cohomol- 
ogy with values in a holomorphic vector bundle on Z in terms of kernels and 
cokernels of differential operators on X. This procedure was developed in a 
representation-theoretic context by Schmid || and independently by Penrose 
as part of his "Twistor Programme" . (A cohomological interpretation of 

Mathematics Subject Classification. Primary 32L25; Secondary 22E46, 53C28. 



1 



Penrose's work did not appear until [EFW|, which considers the transform 
only in the holomorphic category.) 

Let the set of points in X incident with each z G Z be contractible and 
of dimension d. Let V — ► Z be a holomorphic vector bundle. Then under 
conditions which appear below there is a spectral sequence 

E™ = r(X,V m ) =^> H***(Z,V). 

where the V Jhq are certain complex vector bundles over X and the maps at 
the E\ level are first order differential operators. Our main result is that in 
these circumstances one also has a spectral sequence 

E™ = r c (x, V m ) => H^- d (Z, V). 

where the "c" subscripts refer to compactly supported sections and cohomol- 
ogy. 

We also consider the "Serre duality" pairing between cohomology and 
compactly supported cohomology and the resulting pairing between the ker- 
nels and cokernels of differential operators arising from the transform. 



2 Involutive structures and cohomology 

We recall briefly the results about involutive structures and their cohomology 
that we will need. Details can be found in |BES|1 . We then define the com- 



pactly supported involutive cohomology. Our conventions are that TM, 8 M 
will always refer to the complexified tangent bundle and the k-th exterior 
power of the complexified cotangent bundles of the real manifold M. In par- 
ticular, 8 M = £° refers to the trivial complex line bundle whose sections are 
complex-valued functions on M. We omit the "M" and write simply £ k , etc, 
if there can be no confusion as to which manifold is intended. 

Definition 2.1. An involutive structure on the smooth manifold M is a 
complex sub-bundle T 0,1 C TM such that [T°' l ,T 0,1 ] C T 0,1 (meaning that 
the space of smooth sections is closed under Lie bracket). Define the vector 
bundle £ 1,0 C £ x to be the annihilator ofT ' 1 and define £ 0,1 by the exactness 
of 

_ £i>° _ s 1 -> 8 ' 1 -> 0. 
We write £ p ' q = A p £^° ® A 9 ^ ' 1 . 
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Definition 2.2. A complex vector bundle V — > M is compatible with the 
involutive structure £ (or S- compatible) if there is defined a linear operator 

8 : T(M,V) -> T(M, V ® S ' 1 ) 

such that 

B(fs) = fd(s) + (df)s, Wf e £(M), s e r(M, v) 

and such that the extension to 

B : r(M, V <g> £°' q ) -> r(M, V <g> 

satisfies B 2 = 0. 

Definition 2.3. Given an S-compatible vector bundle V — > M define the 
involutive cohomology Hg(M,V) to be the cohomology of the complex 

r(M, i> r(M, f: - 1 ® y) 4 r(M, £ - 2 ® y) 4 • • • 

Writing T C (M, V) for the compactly supported smooth sections ofV, etc, we 
define the compactly supported involutive cohomology if| c (-^f, V) to be the 
cohomology of the complex 

r c (M, v) ^ r c (M, s ' 1 ® v) A r c (M, £ - 2 ® v) i> • • • 

We will be using the following examples of involutive structures: 

1. An involutive structure T 0,1 C TM is a complex structure iff TM = 
T ' 1 ©T ' 1 . In this case the compatible vector bundles are the holomor- 
phic ones and the involutive cohomology is the Dolbeault cohomology. 
The compactly supported cohomology is also the usual compactly sup- 
ported Dolbeault cohomology. 

2. If T 0,1 nT ' 1 = {0} then the involutive structure is a CR-structure. This 
is the structure acquired by a real hypersurface in a complex manifold. 
Our definition is wider than most since it includes "higher codimension" 
cases and includes also complex manifolds. The involutive cohomology 
is often known in this case as Bb cohomology (the "b" standing for 
boundary). 
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Let rj : F — > Z be a fibre bundle. Then C 1 ' = i]*(S^) is an involutive 
structure on F. Bundles on F which are pull-backs of bundles on M are 
compatible. Suppose the fibres of 77 have finite-dimensional de Rham 
cohomology. Then the k-th cohomology of the fibre defines a vector 
bundle 7i k — ► Z. The involutive cohomology in this case (often called 
"relative de Rham cohomology") is fiber- wise de Rham cohomology, 
parametrised by Z. To be precise, we have 

H P (F }V *V) = T(Z,V®H P ). 



The principal calculational tool of ||BKS|| concerns the situation where 



one has two involutive structures A 1 ' C S 1,0 on a manifold M. Defining 
B 1 := £ ' /A ,0 we also have a short exact sequence 

_> B l - ^O' 1 ^ ^ 0. 

Using this to filter the complex for computing H^(M, V) we obtain: 

Proposition 2.4. Let A 1,0 C S 1 ' be two involutive structures on the man- 
ifold M and let V — > M be an A-compatible vector bundle. Then V is also 
S- compatible, and there is a spectral sequence 

= H q e (M, B p ®V)^ H p / q (M, V) 

where B p := A P B\ 

Taking instead the compactly supported cohomology the same filtration 
gives a spectral sequence 

E{' q = Hl c (M, B p <g> V) =^ H p + c q {M, V) 

The proof is straightforward (see ||BES|| for details) and the compactly 
supported case is completely analogous. 

3 The Penrose transform 

We summarise the Penrose transform here as presented in [|BKS|| in a simple 



case. (See also [pD| , [IJ.) The initial data are a double fibration of smooth 



oriented manifolds 



F 



X 

(1) 



with the following properties. 
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1. Z has an involutive structure Q which is a CR-manifold. (We recall 
that our definition includes the possibility that Z is complex.) 

2. The maps r\ and r are fiber-bundle projections such that r has compact 
complex fibers. 

3. The map r\ embeds the fibres of r as holomorphic submanifolds of Z . (A 
submanifold of a CR-manifold is holomorphic if the involutive structure 
of Z restricts to give a complex structure on the submanifold.) 

In this situation, we can endow F with the involutive structure 8 defined by 
S 1 ' being the annihilator of the r-vertical vectors that are of type (0, 1) with 
respect to the complex structure on the fibres of r. 

Let V — > Z be a Q-compatible vector bundle on Z. The Penrose trans- 
form proceeds in three stages. 



Step 1 Define an involutive structure A on F by A 1,0 = r]*Q 1,0 . Then 
Q-cohomology on Z and ^4-cohomology on F can be related as follows. In- 
troduce first the involutive structure C 1 ' = on F. This is exactly the 
third example in §^|. On F we have 

O-^i.n-.c 1 ' -^ ' 1 -^ 

and the standard spectral sequence of Proposition |22§ gives 

E™ = H q c (F, rfV® 7]*Q°' P ) H p / q (F, V *V). 

Using the identification of He in we obtain 

E™ = H P Q (Z,V®H 9 ). 

A particular case of importance is where the fibres of r] are contractible and 
we deduce that 

H p Q (Z, V) S H p A (Z, V) 
with the map being given by pull-back of a representative form. 



Step 2 We define B 1 — > F by B 1 := S 1,Q / A 1 ' and employ the standard 
spectral sequence of Proposition ^]4| to obtain 

E p ' q = H q (F } V *V ® B p ) =^ H p A +q (F, rfV). 
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Step 3 We identify the £-cohomology groups appearing in the previous 
step. The £-cohomology is the Dolbeault cohomology of the fibres of r 
parametrised over X. Let us assume that a complex ^-compatible vector 
bundle E — > F is such that the dimension of the Dolbeault cohomology 
H p {t~ 1 {x), E\ t _ ! ) is constant as x £ X varies. Then this cohomology de- 
fines a vector bundle t$E — > X. (It is in fact the p-th direct image of the 
sheaf of smooth sections of E holomorphic on each fibre of r.) Then 

H p £ (F,E) = r(X,T*E). 

Defining 

v p « = T*(n*v®B*) 

we thus have 

E™ = T{X,V m )^H%\FXV). 

The maps are first order differential operators. 

Combining the above steps in the case where r\ has contractible fibres we 
arrive at the Penrose transform which is the spectral sequence 

E^ = r(X,V M )=>H^{Z,V). 

4 The compactly supported transform 

We retain the setting of §[| supposing also that the fibers of r\ have finite 
dimensional compactly supported de Rham cohomology. We proceed by 
analogy with the real Penrose transform, using the fact that the spectral 
sequence of Proposition ^[4] is valid also for compactly supported cohomology. 
In the first step we will need to identify the compactly supported cohomology 
of the involutive structure C on F. 

Lemma 4.1. Consider the involutive structure C 1,0 = r]*(S^) on F (as in 
Step 1 for the real Penrose transform). For V — ► Z a vector bundle and 
k>0, 

H* jC (F, rj*V) =* r c (Z, H k c g) V) 

where 7i k is the bundle whose fiber over z £ Z is the k-compactly supported 
de Rham cohomology of ri~ l (z). 

Proof. We will use only the case where the fibres of r\ are contractible in 
this paper, when one can apply fiber by fiber the homotopy formula for 
compactly supported cohomology (see e.g. |[BT| , §4]). (The general case then 
follows from a "Cech de Rham complex" argument.) □ 
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Step 1 Following Step 1 of the Penrose transform in the compactly sup- 
ported case and using the above Lemma to identify the C-cohomology we 
obtain the pull-back spectral sequence: 

E™ = H p Qc (Z, n q c ®V)^ H p + c q {F, n*V). 

If the fibers of rj are contractible this immediately converges and we get 

H k QtC (Z,V)^H k +f(F, v *V) 

where d is the dimension of the fibres of 77. Choose p G T(F,C 0,d ) which 
represents "1" in the compactly supported top-degree cohomology of each 
fibre. The map is given in this case by pull-back of forms followed by wedging 
with p. 



Step 2 We use the standard spectral sequence of Proposition \L4$ for com- 
pactly supported cohomology to obtain 



E{' q = Hl c (F, ® r)*V) =► H£(F, rj*V) 



Step 3 We identify fl| >c (F, rfV <g> B p ) = V C (X, V Ptq ) where V p , q are exactly 
the same bundles as arise in the standard transform. This follows almost 
immediately from the corresponding fact for the usual transform. When the 
fibres of 77 are contractible, combining the steps proves the following. 

Theorem 4.2. In the situation of §TJ in the case where the fibres of r] are 
contractible there is a spectral sequence 

e™ = r c (x, v M ) =► H p +r d (z, V). 

The vector bundles that appear are exactly those for the standard transform 
and the differential operators in the spectral sequence are the same as those 
that appear in the non-compactly supported case, but acting between compactly 
supported sections of the relevant bundles. 



5 The bilinear pairing 

For an involutive structure A, define to be the top exterior power of .4. 1 ' 
(by analogy with the definition of the holomorphic canonical bundle on a 
complex manifold). 
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Definition 5.1. On a manifold F with involutive structure A and compatible 
vector bundle V , let k + I = Rank .A ' 1 . The natural bilinear pairing between 
involutive cohomology and compactly supported involutive cohomology on F 

J : HX C (F, V) x H l A (F, V* ® k a ) -> C, 

is that given by wedge product of representative forms (combined with con- 
traction between the vector space V and its dual V*) followed by integration. 

When F is a complex manifold and Q is the complex structure this is 
the Serre duality pairing that, when F is compact, identifies the (necessarily 
finite-dimensional) cohomology spaces as mutually dual. 

Let F now be the correspondence space for the Penrose transform as 
previously. We have also on F the corresponding pairing for ^-cohomology 
and a compatible vector bundle E: 

J : Hl c (F,E) x H r e (F,E* ® k £ ) - C. 

whenever q + r = Rank(£ 0,1 ) (which is the complex dimension of the fibres 
ofr). 

Lemma 5.2. Let k t be the line bundle on F which restricts to each fibre 
of t to be the canonical bundle of that fibre. (Recall that the fibres of r are 
naturally complex manifolds.) Then 

k £ = r*(A^ p ) <g> k t . 

(Here A^ p denotes the line bundle of complex-valued top-degree forms on X.) 

Proof. In the circumstances of the Penrose transform as we have been dis- 
cussing, there is a short exact sequence of vector bundles on F 

-> r*E\ -> 5 1 ' -> E 1 / -> 

where E^'° denotes the vector bundle of forms of type (1, 0) in the complex 
structure of the fibres of r (so that £*'° = S ' 1 ). The result follows by taking 
top exterior powers. □ 

Proposition 5.3. Consider the situation in the Penrose transform where we 
have a S-compatible vector bundle E — > F such that 

HI(F, E) = F(X, rlE), H r £iC (F, E* ® k £ ) = F C (X, r^E* ® k £ )). 
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When q + r = Rank(£ 0,1 ) we can identify 

and the pairing on F for £ cohomology is given by 

[ : T C (X, r*E) x T(X, (r«f?)* ® A^ p ) - C, 
Jx 

where we are contracting the vector bundle t^E with its dual and integrating 
the resulting top- order form over X . 

Proof. Note that by the preceding Lemma, 

E* <g> k £ = E* <g> k t <g> r*A^ p 

and so 

tI(E*®k £ ) = t:{E*®k t )®^. 
For each fibre r" 1 ^ the cohomology groups 

H\t- 1 x,E) and H^t^x, E* <g> k t ) 
are Serre-dual and so we can identify 

® k £ ) = (r«f?)*. 

Now split the pairing integral on F into a fibre integral, which is precisely 
the Serre duality pairing, followed by an integral over the base. □ 

Proposition 5.4. Let 

E™ = r c (X,V p , q )^H^(F,r)*(V)) 

and 

E\ s,t = T(X, (V* <g> K Q ) a>t ) H s A +t (F, V *(V* <g> 

6e t/ie spectral sequences for the Penrose transform as discussed above. There 
is a pairing for r > 1 

£™ x eT'* -> C, p + s = Rank( J B 1 ), g + t = Rank^ ' 1 ) 

which for r = 1 is the pairing for £- cohomology on F and which converges 
to the A-cohomology pairing on F. 
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Proof. Note first that t]*kq = The proof uses the fact that the flltrations 
of the spectral sequences are induced by a sub-bundle of the A ' 1 . One can 
check directly that j F descends to the corresponding terms of the r-level of 
the spectral sequences. □ 

A similar analysis holds for the pull-back stage of our double fibration 
where instead of the involutive structure S 1,0 we consider the involutive struc- 
ture C 1,0 . The outcome in the case where the fibres of 77 are contractible is 
that the pairing 

H k Q ; c d (Z,V)xH l Q (Z,V*®K Q )^C 

(where k + l — d = Rank Q ' 1 and d is the dimension of the fibres of r) pulls 
back to give the pairing 

HX c (F,v*(V)) x H l A (F, V *(V*) ® KA ) -> C. 

6 Examples 

6.1 Euclidean space R 3 

Let X = R 3 and let Z be the total space of the holomorphic tangent bundle 
of CPi, thought of as the parametrisation space of oriented straight lines 
in X. One obtains a double fibration where F is the space of "points on 
oriented lines in R 3 " . 

Let G be the double cover of the group of Euclidean motions of X = R 3 . 
We realise G as 

G = {(A, B) I A e SU(2), B is a 2 x 2 trace- free hermitian matrix} 

acting on X = R 3 = the space of trace-free Hermitian 2x2 matrices x 
according toxi-> Ax A* + B. 

As a homogeneous space Z = G/L where 

-{((^°^(o-°,))^— }■ 

For n G Z, A G C define 0(n, A) to be the G-homogeneous holomorphic 
line bundle on Z associated to the character e ~ me ~ Xz Q f L. Taking Q to be 
the complex structure on Z, so that the involutive cohomology is the usual 
Dolbeault cohomology, it is easy to check that k q = 0(— 4,0). 
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The Penrose transform gives isomorphisms [[I]] 

H^Z, 0(-2, A)) Ker(A + 2A 2 ) 

H 2 Q {Z, 0{-2, A)) Coker(A + 2A 2 ) 

where A denotes the Laplacian mapping the space of smooth functions on 
M 3 to itself. 

The fibres of r] in this case are 1-dimensional, and so we immediately 
obtain the following. 

Theorem 6.1. Let A c denote the mapping from the space of compactly sup- 
ported smooth functions to itself given by the Laplacian. Then 

H° Q>C (Z, 0(-2, A)) Ker(A c + 2A 2 ) 

H^(Z, 0{-2, A)) Coker(A c + 2A 2 ) 

(The first observation is trivial since both sides are zero.) 

The "Serre duality pairing" on Z 

H^Z, 0(-2, A)) x H\(Z, 0(-2, -A)) - C 

translates into the pairing 

UM» [ fg 

where / G Ker(A + 2A 2 ) and g G Coker(A c + 2A 2 ). 
6.2 A CR example 

This is Penrose's original transform, restricted to real Minkowski space (see 

On C 4 with coordinates z\,...,z± define the pseudo-hermit ian form $ by 

z) = ZiZ 3 + Z2Z4 + Z S Zi + z 4 z 2 . 

Let 7 denote the projective line z% = Z4 = and let Z be the 5-dimensional 
CR-manifold 

Z = {z G CP 3 |$(z,z) = 0} \ J. 

The space of complex projective lines which lie in Z can be identified with 
"Minkowski space" X = R 4 with Lorentzian metric. 

Taking Q to be the CR-structure on Z, there are Q-compatible line bun- 
dles 0(n),n G Z which are the restrictions of the usual holomorphic line bun- 
dles on CP3. We have kq = 0{— 4). The fibres of 77 are again 1-dimensional 
and so we have a "dimension shift" of one for the compactly supported co- 
homology. 



11 



6.2.1 The case of 0(-2) 

The Penrose transform gives an isomorphism 

H\{Z,0{-2)) KerD 

where □ is the wave operator associated to the Lorentz structure on M 4 
mapping the space of smooth functions to itself. 

We deduce immediately that the compactly supported transform gives 
isomorphisms 

H QjC (Z,O(-2))^Kera c 
H^Z, 0{-2)) ^ Coker D c 

where D c denotes the wave operator mapping the space of compactly sup- 
ported smooth functions to itself. The first isomorphism is trivial, both sides 
being zero. 

The "Serre duality pairing" on Z 

H^Z, 0{-2)) x H^Z, 0{-2)) -> C 

translates into the pairing 

UM)» [ fa 

where / G Ker □ and g G Coker D c . 

6.2.2 The case of C(-l) and C(-3) 

We will not consider this case in detail. The Penrose transform gives 

if£(Z, 0(-3)) KerZT 

where D~ is the Dirac operator from smooth sections of the spin bundle S~ 
to the other spin bundle S + . 

The compactly supported transform gives 

Hl c (Z,0(-l))^ Coker £>+ 

where is the Dirac operator from compactly supported smooth sections 
of the spin bundle S + to compactly supported smooth sections of S~ . 
The Serre duality pairing between these groups becomes the pairing 

Jm. 4 

where a G KerD - and [j3] G Coker D + and e is the complex bilinear skew 
form on S~ . 
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6.3 The case of 0(-4) and O{0) 

We recall that the smooth complex 2-forms A 2 on X = M 4 split as a direct 
sum of self-dual and anti-self-dual: 

A 2 = A+ © Ai. 

The Penrose transform gives 

H^Z, 0(-4)) Ker{d : A 2 _ -> A 3 ). 
The Penrose transform on O(0) gives 

Ker(d : A 1 -> A 2 ) 



ffi(Z,O(0)) 



Im d : A -> A 



Hl(Z, 0(0)) - A - 



Imd: A 1 -> A^ 
and so we can immediately deduce that 



, NN _ Ker(d: Al -> A 2 c ) 
r °^°(°))^ W:A^aT 



1^(2,0(0))- X 



Imd : Ai -> A 2 



The cohomology group on the left of the first statement is clearly zero and 
so we deduce that a compactly supported complex-valued 1-form with the 
self-dual part of its exterior derivative vanishing is necessarily the exterior 
derivative of a compactly supported function. 
The "Serre duality pairing" on Z 

Hl c (Z,O(0))xH l Q (Z,O(-A))^C 

translates into the pairing 

<JM» [ (f,9) 
where /, g are anti-self-dual 2-forms and (/, g) is the usual bilinear form. 



6.4 Odd-dimensional hyperbolic spaces 

In ||BD|| a twistor correspondence and Penrose transform for X = hyperbolic 
space of dimension 2n + 1 is described. It is equivariant with respect to G — 
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Spin (2n + 1, 1). The space Z is an open orbit in the isotropic Grassmanian 
of complex n-planes in C 2n+2 . There are G- homogeneous holomorphic line 
bundles 0(n, A), n G Z, A G C on Z. 

The smooth Penrose transform gives isomorphisms between Dolbeault 
cohomologies and kernels and cokernels of operators on H 2n+1 . (We omit the 
"Q" for the involutive structure which is the complex structure on Z.) 

H n i n+ 11 ^ ^_ 2n ^ A ^ Ker(A - (A 2 - n 2 )) 

ff^+^Z, 0(-2n, A)) Coker(A - (A 2 - n 2 )) 
# n (Z, C(-2, A)) Ker(A - (A 2 - n 2 )) 
H n+1 (Z, 0(-2, A)) Coker(A - (A 2 - n 2 )) 

where A denotes the hyperbolic Laplacian defined from the space of smooth 
functions on H 2n+1 to itself. 

Similar isomorphisms hold for the compactly supported Dolbeault coho- 
mologies, with a shift in dimension by 1 (which is again the dimension of the 
fibres of if): 

n(n+l) 1 

H c 2 (Z,0{-2n,X)) ^Ker(A c - (A 2 - n 2 )) 

re(n+l) 

H c 2 (Z,0{-2n,X)) Coker(A c - (A 2 - n 2 )) 

0(-2, A)) Ker(A c - (A 2 - n 2 )) 

tf c n (Z, C(-2, A)) Coker(A c - (A 2 - n 2 )) 

where A c denotes the hyperbolic Laplacian from the compactly supported 
smooth functions to itself. 

In this = 0{— 2n — 2, 0) on Z which has complex dimension 

n(n + 3)/2 and so we have Serre-duality pairings 

E n c {Z, 0(-2, A)) x ff^fZ, 0{-2n, -A)) -> C 

n(n + l) 

/£T^(Z,e>(-2n,A)) x H n (Z,0(-2,-\))x 
which both become the pairing 

Coker(A c - (A 2 - n 2 )) x Ker(A - (A 2 - n 2 )) -> C 
induced by multiplication of functions and then integration. The situation 



with the cohomology giving rise to eigenspinors of the Dirac operator in pD|| 
is very similar. 
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